Orthogonal Rank-One Matrix Pursuit for Matrix Completion

Low rank modeling has found applications in a wide range of machine learning and data mining tasks, such as matrix completion, dimensionality reduction, compressed sensing, multi-class and multi-task learning. Recently, significant efforts have been devoted to the low rank matrix completion problem, as it has important applications in many domains including collaborative filtering, Microarray data imputation, and image inpainting. Many algorithms have been proposed for matrix completion in the past. However, most of these algorithms involve computing singular value decomposition, which is not scalable to large-scale problems. In this paper, we propose an efficient and scalable algorithm for matrix completion. The key idea is to extend the well known orthogonal matching pursuit from the vector case to the matrix case. In each iteration, we pursue a rank-one matrix basis generated by the top singular vector pair of the current approximation residual and fully update the weights for all rank-one matrices obtained up to the current iteration. The computation of the top singular vector pair and the updating of the weights can be implemented efficiently, making the proposed algorithm scalable to large matrices. We further establish the linear convergence of the proposed iterative algorithm. This is quite different from the existing theory for convergence rate of orthogonal greedy algorithms. A linear convergence rate is achieved due to our construction of matrix bases. We empirically evaluate the proposed algorithm on many real-world datasets, including the largest publicly available benchmark dataset Netflix as well as the MovieLens datasets. Results show that our algorithm is much more efficient than state-of-the-art matrix completion algorithms while achieving similar or better prediction performance.